Skip to content

Mega MoE: optional per-token weights for fused shared experts - #56

Draft
gau-nernst wants to merge 1 commit into
vllm-project:devfrom
gau-nernst:megamoe-shared-expert-weights
Draft

gau-nernst wants to merge 1 commit into
vllm-project:devfrom
gau-nernst:megamoe-shared-expert-weights

Conversation

@gau-nernst

@gau-nernst gau-nernst commented Nov 9, 2026 •

Copy link
Copy Markdown

Summary

Adds shared_expert_weights, an optional per-token weight on the fused shared expert, to fp8_fp4_mega_moe (SM100, SwiGLU) and bf16_mega_moe.

Qwen MoE models gate the shared expert per token:

$$y = \sum_{i \in \text{top-}k} w_i, E_i(x) + \sigma(w_s^\top x), S(x)$$

Mega MoE can fuse a shared expert today, but only as the unweighted + S(x), so these models have to run the shared expert separately. vllm-project/vllm#66659 uses this PR to fuse it.

Design

  • API. shared_expert_weights: Optional[torch.Tensor] = None, FP32, (num_tokens,), contiguous, on the device of y. The caller computes the weight, e.g. sigmoid(x @ w_s), so the kernel stays agnostic to the gating function. None keeps the current behavior. It is the last argument everywhere, including the op schemas, so existing callers are unaffected. The checks are inline in csrc/apis/mega_moe.hpp, next to the other optional inputs.
  • Kernel. Routed rows already multiply the SwiGLU output by their top-k weight in the L1 epilogue, before the L2 input is quantized; shared rows used a constant 1. Shared rows now load the token's weight there instead. Since g * W2(h) = W2(g * h), the math is the same as scaling the output.
  • Compile-time flag. The weights path is behind a template flag kWithSharedExpertWeights (like kWithPostExpertNorm), so None compiles the same kernel as before. A static_assert requires fused shared experts when the flag is set.
  • CUDA graphs. The tensor needs a stable address, like y.
  • Coverage. Only the SM100 FP8/FP4 SwiGLU kernel and the BF16 kernel (both weight layouts) support it. The SM90 and SiTU kernels reject it; NVFP4 and FP4xFP4 Mega MoE don't take it yet.

Numerics: the kernel quantizes g * h into the L2 input, not g * quant(h), the same treatment routed experts get from their top-k weights.

Tests

tests/test_mega_moe.py gains --shared-expert-weights:

  • The PyTorch reference (--emu-reference) and the legacy baseline weight the shared expert by it.
  • For exact inputs, the weights are random powers of two, so the exact comparison still holds.
  • The CUDA graph check negates the weights in its second input set, so a replay must pick up new weights written into the same tensor.
  • --sweep gains six cases with weights. They cover zero and one token, multiple shared experts, the hot / empty / masked routings, and up to 8192 tokens. These cases are skipped on SM90.

Results on an 8x B300 node, --emu-reference, for the six sweep shapes with weights, run directly at 1 and 8 ranks with fp8xfp4 and fp8xfp8, plus a Qwen-shaped case (hidden 2560, 512 experts, intermediate 640, top-10, 4096 tokens):

  • Accuracy passes in all 24 runs: max_token_diff ≤ 2.8e-5, and 0 for the 0- and 1-token cases.
  • 1 rank: all runs pass.
  • 8 ranks: the bitwise CUDA graph replay check fails in 8 of 12 runs, at the first or second replay. This also happens on dev without this PR and without weights: the first --sweep case at 8 ranks fails it, and so do the 1-token and Qwen-shaped cases with no weights. It is not caused by this change.
  • The full --sweep stops at the first failure, and it fails at the same point on dev and on this branch with identical numbers:
    • at 8 ranks, the graph check above;
    • at 1 rank, (8192, 7168, 128, 65, 8, 2, 10, 'masked') with max_token_diff=1.79e-4 > 1e-4.

Also tested:

  • An earlier standalone test passes for bf16xbf16 and fp8xfp4 at 1 and 8 ranks. It checks power-of-two weights exactly with only the shared expert active, a sigmoid-gated linearity check with routing, every BLOCK_M the heuristic picks, an empty rank, and rejection of SiTU and misshaped weights.
  • tests/test_bf16_mega_moe_trtllm.py passes.
  • After the last kernel change (the shared-weight reload got the same WG_BLOCK_M guard as the routed reload), the weighted cases at 1 rank (fp8xfp8, fp8xfp4) and the standalone test at 1 and 4 ranks (bf16xbf16, fp8xfp8, fp8xfp4) pass again.
  • Not run: the legacy-baseline path. DeepEP and TileLang were not installed.

Performance

Fused vs. unfused gated shared expert

Without this PR, a gated shared expert cannot be fused. It runs as a separate BF16 MLP next to a MegaMoE that runs only the routed experts, as in vllm-project/vllm#64315. The comparison uses a Qwen MoE shape (hidden 2560, 512 experts with intermediate 640, top-10, one shared expert of 640), MXFP8 (fp8xfp8), EP4 on 4x B300, as in the vLLM DP4 + EP4 serve. Times are in µs.

  • MegaMoE: tests/test_mega_moe.py per-rank kernel time, range over two runs. The unfused arm is dev with --num-shared-experts 0. The fused arm is this PR with --num-shared-experts 1 --shared-expert-weights.
  • Shared MLP: BF16 gate_up GEMM, fused SiLU·mul, BF16 down GEMM, and a fused y += sigmoid(g) * s. Measured on one GPU with a CUDA graph.
  • Excluded: the gate GEMV and its sigmoid (6.3 - 13.7 µs), which run in both arms.
Tokens / rank MegaMoE, routed only + shared MLP Unfused total MegaMoE, fused (this PR) Change
32 137.0 - 137.8 9.9 147.3 135.2 - 138.5 -7%
64 146.2 - 146.8 10.2 156.7 146.2 - 147.0 -6%
128 147.5 10.7 158.2 151.0 -5%
256 149.5 - 151.5 11.4 161.9 149.5 - 152.8 -7%
1024 188.0 - 188.8 17.0 205.4 189.5 - 192.0 -7%
4096 521.5 - 522.2 44.0 565.9 530.8 - 531.2 -6%

Fusing the gated shared expert adds 0 - 3.5 µs to MegaMoE at decode sizes and about 9 µs at 4096 tokens per rank. It removes a 10 - 44 µs MLP. This is an upper bound: in vLLM the unfused MLP runs on a side stream and partly overlaps the router. vllm-project/vllm#66659 measures the end-to-end effect.

No cost without weights

tests/test_mega_moe.py, 8x B300, EP8, with one fused (unweighted) shared expert. Mean kernel time in µs, range over two runs:

Case dev this PR, None this PR, weights
Qwen shape, fp8xfp4, 128 tok/rank 80.0 - 80.8 78.8 - 81.2 80.1 - 82.0
Qwen shape, fp8xfp4, 4096 tok/rank 578.4 - 580.0 578.8 - 580.2 578.9 - 579.5
Qwen shape, bf16, 4096 tok/rank 769.0 - 770.4 770.1 - 770.2 770.8 - 771.4

For the None path, I compared the SASS of six kernel instantiations with dev. The opcode sequences are identical. The only difference is kernel-parameter offsets, which shift by 8 bytes because of the new pointer argument. The weights code is behind if constexpr, so the None kernel does not change.

Not done yet (draft)

  • The legacy-baseline comparison with weights (needs DeepEP + TileLang).

🤖 Generated with Claude Code

Add `shared_expert_weights`, an optional FP32 `(num_tokens,)` tensor, to
`fp8_fp4_mega_moe` (SM100, SwiGLU) and `bf16_mega_moe` (including the TRT-LLM
weight layout). For shared-expert rows, the L1 epilogue multiplies the
activation output by the token's weight, at the same place routed rows apply
their top-k weight. This lets models that gate the shared expert per token,
e.g. Qwen MoE's `sigmoid(shared_expert_gate(x)) * shared_expert(x)`, fuse it.

The weights path is a template flag (`kWithSharedExpertWeights`), so
`None` instantiates the same kernel as before: the only SASS difference is
the kernel-parameter offsets shifted by the new pointer argument.

The SM90 and SiTU kernels reject the weights; NVFP4 and FP4xFP4 Mega MoE do
not take them yet. `tests/test_mega_moe.py` gains `--shared-expert-weights`,
sweep cases with the weights, and covers them in the reference, the legacy
baseline and the CUDA graph check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gau-nernst
gau-nernst force-pushed the megamoe-shared-expert-weights branch from 7d2eaee to ee9be6f Compare November 10, 2026 02:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant